Configuration-preserving preprocessor

ABSTRACT

Methods, systems, and apparatuses, including computer programs encoded on computer readable media, for generating a plurality of tokens from one or more source files. The one or more source files include source code in a first programming language. The source code includes one or more static conditionals that include a conditional expression and branch code that is operative when the conditional expression is true. Various configurations are possible based upon the conditionals. A first static conditional that includes one or more nested static conditionals within the branch code associated with the first static conditional is determined. Each of the one or more nested static conditionals is hoisted to a beginning of the branch code associated with the first static conditional. Each innermost branch code does not contain a static conditional, and each possible configuration is preserved.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/485,410, filed on May 31, 2012, which is incorporated herein byreference in its entirety.

GOVERNMENT RIGHTS

This invention was made with government support under Grant Nos.CNS-0448349, CNS-0615129, and CCF-1017849 awarded by the NationalScience Foundation. The government has certain rights in the invention.

BACKGROUND

Large-scale software development requires effective tool support, suchas source code browsers, bug finders, and automated refactorings. Thisneed is especially pressing for C, since it is the language of choicefor critical software infrastructure, including the Linux kernel andApache web server. However, building tools for C presents a specialchallenge. C is not only low-level and unsafe, but source code mixes twolanguages: the C language proper and the preprocessor. These tools,therefore, need to process C itself and the preprocessor. Thepreprocessor adds facilities lacking from C itself. Notably, fileincludes (#include) provide rudimentary modularity, macros (#define)enable code transformation with a function-like syntax, and staticconditionals (#if, #ifdef, and so on) capture variability. Thepreprocessor is oblivious to C constructs and operates only onindividual tokens. Real-world C code reflects both points: preprocessorusage is widespread and often violates C syntax.

Existing C tools do not process both languages. Rather, they eitherprocess one configuration at a time (e.g., the Cxref source browser, theAstree bug finder, and Xcode refactorings), rely on a single, maximalconfiguration (e.g., the Coverity bug finder), or build on incompleteheuristics (e.g., the LXR source browser and Eclipse refactorings).Processing one configuration at a time is infeasible for large programssuch as Linux, which has over 10,000 configuration variables. Maximalconfigurations cover only part of the source code, mainly due to staticconditionals with more than one branch. For example, Linux' allyesconfigenables less than 80% of the code blocks contained in conditionals. Andheuristic algorithms prevent programmers from utilizing the fullexpressivity of C and its preprocessor.

SUMMARY

In general, one aspect of the subject matter described in thisspecification can be embodied in methods for Methods, systems, andapparatuses, including computer programs encoded on computer readablemedia, for generating a plurality of tokens from one or more sourcefiles. The one or more source files include source code in a firstprogramming language. The source code includes one or more staticconditionals that include a conditional expression and branch code thatis operative when the conditional expression is true. Variousconfigurations are possible based upon the conditionals. A first staticconditional that includes one or more nested static conditionals withinthe branch code associated with the first static conditional isdetermined. Each of the one or more nested static conditionals ishoisted to a beginning of the branch code associated with the firststatic conditional. Each innermost branch code does not contain a staticconditional, and each possible configuration is preserved. Eachinnermost branch code does not contain a static conditional and eachpossible configuration is preserved. Other implementations of thisaspect include corresponding systems, apparatuses, and computer-readablemedia configured to perform the actions of the method.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects,implementations, and features described above, further aspects,implementations, and features will become apparent by reference to thefollowing drawings and the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features of the present disclosure will becomemore fully apparent from the following description and appended claims,taken in conjunction with the accompanying drawings. Understanding thatthese drawings depict only several implementations in accordance withthe disclosure and are, therefore, not to be considered limiting of itsscope, the disclosure will be described with additional specificity anddetail through use of the accompanying drawings.

FIG. 1 illustrates an abstract syntax tree (AST) in accordance with anillustrative implementation.

FIG. 2 shows the cumulative distribution of subparser counts per FMLRiteration for the x86 Linux kernel under different optimization levelsin accordance with various illustrative implementations.

FIG. 3 shows the cumulative latency distribution across compilationunits of the constrained kernel of one illustrative implementation andTypeChef when ran on an off-the-shelf PC.

FIG. 4 plots the breakdown of the latency of a tested implementation.

FIG. 5 is a block diagram of a computer system in accordance with anillustrative implementation.

Reference is made to the accompanying drawings throughout the followingdetailed description. In the drawings, similar symbols typicallyidentify similar components, unless context dictates otherwise. Theillustrative implementations described in the detailed description,drawings, and claims are not meant to be limiting. Other implementationsmay be utilized, and other changes may be made, without departing fromthe spirit or scope of the subject matter presented here. It will bereadily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, and designed in a wide variety ofdifferent configurations, all of which are explicitly contemplated andmade part of this disclosure.

DETAILED DESCRIPTION

C compilers, such as gcc, process only one variant of the source code ata time. They pick the one branch of each static conditional that matchesthe configuration variables passed to the preprocessor, e.g., throughthe -D command line option. Different configuration variable settings,or configurations, result in different executables, all from the same Csources. In contrast, other C tools, such as source browsers, bugfinders, and automated refactorings, need to beconfiguration-preserving. They need to process all branches of staticconditionals and, for each branch, track the presence condition enablingthat branch. This considerably complicates C tools besides compilers,starting with preprocessing and parsing.

FIG. 1 illustrates an implementation of configuration-preservingpreprocessing and parsing on a simple example from the x86 Linux kernel(version 2.6.33.3, which is used throughout this description). Table 1below shows the original source code, which utilizes the three mainpreprocessor facilities: an include directive on line 1, macrodefinitions on lines 3 and 4, and conditional directives on lines 10 and14. The code has two configurations, one when CONFIG_INPUTMOUSEDEV_PSAUXis defined and one when it is not defined. After preprocessing, shown inTable 2, the header file has been included (not shown) and the macroshave been expanded on lines 6, 7, and 10, but the conditional directivesremain on lines 5 and 9.

FIG. 1 illustrates an abstract source tree (AST) of the example shown inTable 2 in accordance with an illustrative implementation. The AST 100contains both configurations with a static choice node 102 correspondingto the static conditional on lines 5-9 in Table 2.

TABLE 1 1 #include “major.h” // Defines MISC_MAJOR to be 1 2 3 #defineMOUSEDEV_MIX 31 4 #define MOUSEDEV_MINOR_BASE 32 5 6 static intmousedev_open(struct inode *inode, struct file *file) 7 { 8  int i; 9 10#ifdef CONFIG_INPUT_MOUSEDEV_PSAUX 11  if (imajor(inode) == MISC_MAJOR)12   i = MOUSEDEV_MIX; 13  else 14 #endif 15   i = iminor(inode) −MOUSEDEV_MINOR_BASE; 16 17  return ; 18 }

TABLE 2 1 static int mousedev_open(struct inode *inode, struct file*file) 2 { 3  int 1; 4 5 #ifdef CONFIG_INPUT_MOUSEDEV_PSAUX 6  if(imajor(inode) == 1) 7   i = 31; 8  else 9 #endif 10   i =iminor(inode) − 32; 11 12  return ; 13 }

The complexity of configuration-preserving C processing stems from theinteraction of preprocessor features with each other and with the Clanguage. Table 3 summarizes these interactions. Rows denote languagefeatures and are grouped by the three steps of C processing: lexing,preprocessing, and parsing. The first column names the feature and thesecond column describes the implementation strategy. The remainingcolumns capture complications arising from the interaction of features,and the corresponding table entries indicate how to overcome thecomplications. Blank entries indicate impossible interactions. Invarious implementations, the configuration-preserving preprocessoraddresses all interactions.

TABLE 3 Surrounded by Contain Contain Multiply- Language ConstructImplementation Conditionals Conditionals Defined Macros Other LexerLayout Annotate tokens Preprocessor Macro (Un)Definition Use conditionalAdd multiple entries Do not expand Trim infeasible entries macro tableto macro table until invocation on redefinition Object-Like Expand allIgnore infeasible Expand nested Get ground truth for Macro Invocationsdefinitions definitions macros built-ins from compiler Function-LikeExpand all Ignore infeasible Hoist conditionals Expand nested Supportdiffering argument Macro Invocations definitions definitions aroundinvocations macros numbers and variadies Token Pasting & Apply pasting &stringification Hoist conditionals around Stringification token pasting& stringification File Includes Include and Preprocess under Hoistconditionals Reinclude when guard preprocess files presence conditionsaround includes macro is not false Static Conditionals Preprocess allConjoin presence conditions Ignore infeasible branches definitionsConditional Expressions Evaluate presence Hoist conditionals Presenceorder for non- conditions around expressions boolean expressions ErrorDirectives Ignore infeasible branches Line, Warning, & Treat us PragmaDirectives layout Parser C Constructs Use FMLR Parser Fork and mergesubparsers Typedef Names Use conditional Add multiple entries Forksubparsers on symbol table to symbol table ambiguous names

Layout. The first step is lexing. The lexer converts raw program textinto tokens, stripping layout such as whitespace and comments. Sincelexing is performed before preprocessing and parsing, it does notinteract with the other two steps. However, automated refactorings, bydefinition, restructure source code and need to output program text asoriginally written, modulo any intended changes. Consequently, they needto annotate tokens with surrounding layout-plus, keep sufficientinformation about preprocessor operations to restore them as well.

Macro (un)definitions. The second step is preprocessing. It collectsmacro definitions (#define) and undefinitions (#undef) in a macrotable-with definitions being either object-like:

-   -   #define name body        or function-like    -   #define name (parameters) body

Definitions and undefinitions for the same macro may appear in differentbranches of static conditionals, creating a multiply-defined macro thatdepends on the configuration. Table 4 shows such a macro, BITS_PER_LONG,whose definition depends on the CONFIG_(—)64BIT configuration variable.A configuration-preserving preprocessor can record all definitions inits macro table, tagging each entry with the presence condition of the#define directive while also removing infeasible entries on each update.The preprocessor can also record undefinitions, so that it can determinewhich macros are neither defined nor undefined and thus free, i.e.,configuration variables. Wherever multiply-defined macros are used, theypropagate an implicit conditional. It is as if the programmer hadwritten an explicit conditional in the first place.

TABLE 4   1 #ifdef CONFIG_64BIT 2 #define BITS_PER_LONG 64 3 #else 4#define BITS_PER_LONG 32 5 #endif

Macro invocations. Since macros may be nested within each other, aconfiguration-preserving preprocessor, just like an ordinarypreprocessor, can recursively expand each macro. Furthermore, since Ccompilers have built-in object-like macros, such as _STDG_VERSION_ toindicate the version of the C standard, the preprocessor can beconfigured with the ground truth of the targeted compiler.

Beyond these straightforward issues, a configuration-preservingpreprocessor can handle two, more subtle interactions. First, a macroinvocation may be surrounded by static conditionals. Consequently, thepreprocessor can ignore macro definitions that are infeasible for thepresence condition of the invocation site. Second, function-like macroinvocations may contain conditionals, either explicitly in source codeor implicitly through multiply-defined macros. These conditionals canalter the function-like macro invocation by changing its name orarguments, including their number and values. To preserve thefunction-like invocation while also allowing for differing argumentnumbers and variadics (a gcc extension) in different conditionalbranches, the preprocessor can hoist the conditionals around theinvocation.

Tables 5 and 6 illustrate the latter issue. Table 5 contains a sequenceof tokens on line 10, cpu_to_le32 (val), which either expands to aninvocation of the function-like macro _cpu_to_le32, if KERNEL_ isdefined, or denotes the invocation of the C function cpu_to_le32,if_KERNEL_ is not defined. Table 6 shows the three stages ofpreprocessing the sequence. First, in 6a, the preprocessor expandscpu_to_le32, which makes the conditional explicit but also breaks thenested macro invocation on line 2. Second, in 6b, the preprocessorhoists the conditional around the entire sequence of tokens, whichduplicates (val) in each branch and thus restores the invocation on line2. Third, in 6c, the preprocessor recursively expands _cpu_to_le32 online 2, which completes preprocessing for the sequence.

TABLE 5 1 // In include/linux/byteorder/little_endian.h: 2 #define__cpu_to_le32(x) ((__force __le32)(__u32)(x)) 3 4 #ifdef __KERNEL__ 5 //Included from include/linux/byteorder/generic.h: 6 #define cpu_to_le32__cpu_to_le32 7 #endif 8 9 // In drivers/pci/proc.c: 10_put_user(cpu_to_le32(val), (__le32 __user *) buf);

TABLE 6 1 #ifdef __KERNEL__ 2 __cpu_to_le32 3 #else 4 cpu_to_le32 5#endif 6 (val)     (a) After expansion of cpu_to_le32. 1 #ifdef__KERNEL__ 2 __cpu_to_le32(val) 3 #else 4 cpu_to_le32(val) 5 #endif    (b) After hoisting the conditional. 1 #ifdef __KERNEL 2 ((__force__le32) (__u32) (val)) 3 #else 4 cpu_to_le32(val) 5 #endif     (c) Afterexpansion of __cpu_to_le32.

Token-pasting and stringification. Macros may contain two operators thatmodify tokens. The infix token-pasting operator ## concatenates twotokens, and the prefix stringification operator # converts a sequence oftokens into a string literal. The preprocessor simply applies theseoperators, with one complication: the operators' arguments may containconditionals, either explicitly in source code or implicitly viamultiply-defined macros. As for function-like macros, aconfiguration-preserving preprocessor needs to hoist conditionals aroundthese operators. Table 7 illustrates this for token-pasting: 7a showsthe source code; 7b shows the result of expanding all macros, includingBITS_PER_LONG from Table 4; and 7c shows the result of hoisting theconditional out of the token-pasting.

TABLE 7 1 #define uintBPL_t uint (BITS_PER_LONG) 2 #define uint(x)xuint(x) 3 #define xuint(x) __le ## x 4 5 uintBPL_t *p = ... ;     (a)The macro definitions and invocation. 1 __le ## 2 #ifdef CONFIG_64BIT 364 4 #else 5 32 6 #endif 7 *p = ... ;     (b) After expanding themacros. 1 #ifdef CONFIG_64BIT 2 __le ## 64 3 #else 4 __le ## 32 5 #endif6 *p = ... ;     (c) After hoisting the conditional.

File includes. To produce complete compilation units, aconfiguration-preserving preprocessor can recursively resolve fileincludes (#include). If the directive is nested in a static conditional,the preprocessor can process the header file under the correspondingpresence condition. Furthermore, if a guard macro, which istraditionally named FILENAME_H and protects against multiple inclusion,has been undefined, the preprocessor needs to process the same headerfile again. More interestingly, include directives may contain macrosthat provide part of the file name. If the macro in such a computedinclude is multiply-defined, the preprocessor can hoist the implicitconditional out of the directive, just as for function-like macroinvocations, token-pasting, and stringification.

Conditionals. Static conditionals enable multiple configurations, soboth configuration-preserving preprocessor and parser can process allbranches. The preprocessor converts static conditionals' expressionsinto presence conditions, and when conditionals are nested within eachother, conjoins nested conditionals' presence conditions. As describedfor macro invocations above, this lets the preprocessor ignoreinfeasible definitions during expansion of multiply-defined macros.

However, two issues complicate the conversion of conditional expressionsinto presence conditions. First, a conditional expression may containarbitrary macros, not just configuration variables. So the preprocessorcan expand the macros, which may be multiply-defined. When expanding amultiply-defined macro, the preprocessor can convert the macro'simplicit conditional into logical form and hoist it around theconditional expression. For example, when converting the conditionalexpression

BITS_PER_LONG= =32

from kernel/sched.c into a presence condition, the preprocessor canexpand the definition of BITS_PER_LONG from Table 4 and hoists it aroundthe conditional expression, to arrive at

defined(CONFIG_(—)64BIT)&& 64= =32\

II !defined(CONFIG_(—)64BIT)&& 32= =32

which makes testing for CONFIG_(—)64BIT explicit with the definedoperator and simplifies to

!defined(CONFIG_(—)64BIT)

after constant folding.

Second, configuration variables may be non-boolean and conditionalexpressions may contain arbitrary arithmetic subexpressions, such asNR_(—CPUS<)256 (from arch/x86/include/asm/spinlock.h). Since there is noknown efficient algorithm for comparing arbitrary polynomials, suchsubexpressions prevent the preprocessor from trimming infeasibleconfigurations. Instead, it can treat non-boolean subexpressions asopaque text and preserve their branches' source code ordering, i.e.,never omit or combine them and never move other branches across them.

Other preprocessor directives. The C preprocessor supports fouradditional directives, to issue errors (#error) and warnings (#warning),to instruct compilers (#pragma), and to overwrite line numbers (#line).A configuration-preserving preprocessor simply reports errors andwarnings appearing outside static conditionals, and also terminates forsuch errors. More importantly, it can treat conditional branchescontaining error directives as infeasible and disables their parsing.Otherwise, it can preserve such directives as token annotations tosupport automated refactorings.

C constructs. The third and final step is parsing. The preprocessorproduces entire compilation units, which may contain static conditionalsbut no other preprocessor operations. The configuration-preservingparser processes all branches of each conditional by forking itsinternal state into subparsers and merging the subparsers again afterthe conditional. This way, it produces an AST containing allconfigurations, with static choice nodes for conditionals.

One significant complication is that static conditionals may stillappear between arbitrary tokens, thus violating C syntax, while the ASTmay only contain nodes representing complete C constructs. To parse Cconstructs with embedded configurations, the parser may require asubparser per embedded configuration and may also parse tokens sharedbetween configurations several times. For example, the statement onlines 5-10 in Table 2 has two configurations and requires twosubparsers. Furthermore, line 10 is parsed twice, once as part of theif-then-else statement and once as a stand-alone expression statement.This way, the parser hoists static conditionals out of C constructs in asimilar way compared to the preprocessor hoisting static conditionalsout of preprocessor operations.

This is acceptable for most declarations, statements, and expressions,which have a small number of terminals and non-terminals and thus cancontain only a limited number of configurations. However, if a Cconstruct contains repeated non-terminals, this can lead to anexponential blow-up of configurations and therefore subparsers. Forexample, the array initializer in Table 8 has 2¹⁸ unique configurations.Using a subparser for each configuration is clearly infeasible andavoiding this requires careful optimization of the parsing algorithm.

TABLE 8 1 static int (*check_part[ ])(struct parsed_partitions *) = { 2#ifdef CONFIG_ACORN_PARTITION_ICS 3  adfspart_check_ICS. 4 #endif 5#ifdef CONFIG_ACORN_PARTITION_POWERTEC 6  adfspart_check_POWERTEC. 7#endif 8 #ifdef CONFIG_ACORN_PARTITION_EESOX 9  adfspart_check_EESOX. 10#endif 11  // 15 more, similar initializers 12  NULL 13 };

Typedef names. A final complication results from the fact that C syntaxis context-sensitive. Depending on context, names can either be typedefnames, i.e., type aliases, or they can be object, function, and enumconstant names. Furthermore, the same code snippet can havefundamentally different semantics, depending on names. For example,

T*p;

is either a declaration of p as a pointer to type T or an expressionstatement that multiplies the variables T and p, depending on whether Tis a typedef name. C parsers usually employ a symbol table todisambiguate names. In the presence of conditionals, however, a name maybe both. Consequently, a configuration-preserving parser can maintainconfiguration-dependent symbol table entries and fork subparsers whenencountering an implicit conditional due to an ambiguously defined name.

Implementations of a configuration-preserving preprocessor andcorresponding configuration-preserving parser include features asdescribed above. Two features, however, require further elaboration: thehoisting of static conditionals around preprocessor operations and theconversion of conditional expressions into presence conditions.

Hoisting Static Conditionals. Preprocessor directives as well asfunction-like macro invocations, token-pasting, and stringification mayonly contain ordinary language tokens. Consequently, they areill-defined in the presence of implicit or explicit embedded staticconditionals. To perform these preprocessor operations, variousimplementations of a configuration-preserving preprocessor can hoistconditionals, so that only ordinary tokens appear in the branches of theinnermost conditionals.

TABLE 9  1: procedure HOIST(c, τ)  2:  

 Initialize a new conditional with an empty branch.  3:  C ← [ (c, •) ] 4:  for all a ε τ do  5:   if a is a language token then  6:    

 Append a to all branches in C.  7:    C ← [ (c_(i), τ_(i)a) | (c_(i),τ_(i)) ε C]  8:   else

 a is a conditional,  9:    

 Recursively hoist conditionals in each branch. 10:    B ← [b | b εHOIST(c_(i), τ_(i)) and (c_(i), τ_(i)) ε α ] 11:    

 Combine with already hoisted conditionals. 11    C ← C × B 13:   end if14:  end for 15:  return C 16: end procedure

Table 9 illustrates one example of a HOIST algorithm. It takes apresence condition c and a list of ordinary tokens and entireconditionals τ under the presence condition. Each static conditional C,in turn, is treated as a list of branches:

CL=[(c ₁,τ₁), . . . , (c _(n),τ_(n))]

with each branch having a presence condition c, and a list of tokens andnested conditionals τ₁. Line 3 initializes the result C with an emptyconditional branch. Lines 4-14 iterate over the tokens and conditionalsin τ, updating C as necessary. And line 15 returns the result C. Lines5-7 of the loop handle ordinary tokens, which are present in allembedded configurations and are appended to all branches in C, asillustrated for (val) in Table 6b and for_le## in Table 7c. Lines 8-13of the loop handle static conditionals by recursively hoisting anynested conditionals in line 10 and then combining the result B with C inline 12. The cross product for conditionals in line 12 is defined as:

C×B:=[(c _(i)

c _(j) ,τ, _(i)τ_(j))|(c _(i),τ_(i))εC and (c _(j),τ_(j))εB]

and generalizes line 7 by combining every branch in C with every branchin B.

In one implementation, a configuration-preserving preprocessor usesHOIST for all preprocessor operations that may contain conditionalsexcept for function-like macro invocations. The problem with the latteris that, to call HOIST, the preprocessor needs to know which tokens andconditionals belong to an operation. But different conditional branchesof a function-like macro invocation may contain different macro namesand numbers of arguments, and even additional, unrelated tokens.Consequently, various implementations of the configuration-preservingpreprocessor can use a version of HOIST for function-like macroinvocations that interleaves parsing with hoisting. For each conditionalbranch, it tracks parentheses and commas, which change the parsing stateof the invocation. Once all variations of the invocation have beenrecognized across all conditional branches, each invocation isseparately expanded. If a variation contains an object-like or undefinedmacro, the argument list is left in place, as illustrated in Table 6c,line 4.

One implementation tracks all configurations of an invocation bymaintaining a list of pairs, where each pair is an invocation parsingstate with its presence condition. The invocation state can contain anargument count, parenthesis depth, a parsing result, and whatever stateis needed for parsing. To determine the presence conditions of allconfigurations, an implementation works like this. Starting with oneinvocation parsing state, it reads tokens one-by-one. Each token is usedto update the parsing state of all configurations of the invocation.Before updating one configuration's parsing state, the procedure checkswhether the token's presence condition and the configuration's presencecondition are different (but not mutually exclusive). If they aredifferent, the parsing state is duplicated, and the two duplicatedpresence conditions are (1) the original configuration's presencecondition conjoined with the token's presence condition and (2) theoriginal configuration's presence condition conjoined with the negationof the token's presence condition. This process is repeated with tokensfrom the input until all configurations of the invocation have finishedparsing or found a parsing error. To hoist the conditional around theinvocation, an implementation can create a new conditional. Eachconfiguration is given one branch in the new conditional. Each branch'spresence condition is one configuration's, and the branch contains thetokens that appear in that configuration (i.e., the token's whosepresence condition is not mutually exclusive with the configuration's).Once conditionals have been hoisted around the invocation, thepreprocessor can process the conditional as usual, invoking eachbranch's function-like macro.

Converting Conditional Expressions. To reason about presence conditions,conditional expressions can be converted into binary functions. Invarious implementations conditional expressions can be converted intoBinary Decision Diagrams (BDDs), which are an efficient, symbolicrepresentation of boolean functions. BDDs include support for booleanconstants, boolean variables, as well as negation, conjunction, anddisjunction. On top of that, BDDs are canonical: Two boolean functionsare the same if and only if their BDD representations are the same. Thismakes it not only possible to directly combine BDDs, e.g., when trackingthe presence conditions of nested or hoisted conditionals, but also toeasily compare two BDDs for equality, e.g., when testing for aninfeasible configuration by evaluating c₁

c₂=false.

Before converting a conditional expression into a BDD, theconfiguration-preserving preprocessor can expand any macros outsideinvocations of the defined operator, can hoist multiply-defined macrosaround their sub-expressions, and can perform constant folding. Theresulting conditional expression uses negations, conjunctions, anddisjunctions to combine four types of sub-expressions: constants, freemacros, arithmetic expressions, and defined invocations. Theconfiguration-preserving preprocessor can convert each of thesesub-expressions into a BDD as follows and then combines the resultingBDDs with the necessary logical operations:

-   1. A constant translates to false if zero and to true otherwise.-   2. A free macro translates to a BDD variable.-   3. An arithmetic subexpression also translates to a BDD variable.-   4. defined(M) translates into the disjunction of presence    conditions, under which M is defined. However, if M is free:    -   a. If M is a guard macro, defined(M) translates to false.    -   b. Otherwise, defined(M) translates to a BDD variable.

Just like gcc, Case 4a treats M as a guard macro, if a header filestarts with a conditional directive that tests !defined(M) and isfollowed by #define M, and the matching #endif ends the file. To ensurethat repeated occurrences of the same free macro, arithmetic expression,or defined(M) for free M translate to the same BDD variable, theconfiguration-preserving preprocessor can maintain a mapping betweenthese expressions and their BDD variables. In the case of arithmeticexpressions, it normalizes the text by removing whitespace and comments.

Binary functions, other than BDD, can be used in other implementations.For example, an automated SAT solver or an automated SMT solver can beused to encode binary functions.

The Configuration-Preserving Parser. In various implementations, theinvention can include a configuration-preserving fork-merge LR (FMLR)parser that builds on LR parsing, a bottom-up parsing technique. Torecognize the input, LR parsers can maintain an explicit parser stack,which contains terminals, i.e., tokens, and non-terminals. On each step,LR parsers can perform one of four actions: (1) shift to copy a tokenfrom the input onto the stack and increment the parser's position in theinput; (2) reduce to replace one or more top-most stack elements with anon-terminal; (3) accept to successfully complete parsing; and (4)reject to terminate parsing with an error. The choice of action dependson both the next token in the input and the parser stack. To ensureefficient operation, LR parsers use a deterministic finite control andstore the state of the control with each stack frame.

Compared to top-down parsing techniques, such as LL and PEG, LR parsersare an attractive foundation for configuration-preserving parsing forthree reasons. First, LR parsers make the parsing state explicit, inform of the parser stack. Consequently, it is easy to fork the parserstate on a static conditional, e.g., by representing the stack as asingly-linked list and by creating new stack frames that point to theshared remainder. Second, LR parsers are relatively straight-forward tobuild, since most of the complexity lies in generating the parsingtables, which determine control transitions and actions. Variousimplementations of a configuration-preserving parser can use tablesproduced by an existing parser generator. Third, LR parsers supportleft-recursion in addition to right-recursion, which can be helpful forwriting programming language grammars.

Fork-Merge LR Parsing. Table 10 illustrates one example of a FMLRparsing algorithm. It uses a queue Q of LR subparsers p. Each subparserp :=(c, a, s) has a presence condition c, a next token or conditional a,which is also called head, and an LR parser stack s. Each subparserrecognizes a distinct configuration, i.e., the different presenceconditions p.c are mutually exclusive, and all subparsers togetherrecognize all configurations, i.e., the disjunction of all theirpresence conditions is true. Q is a priority queue, ordered by theposition of the head p.a in the input. This ensures that subparsersmerge at the earliest opportunity, as no subparser can outrun the othersubparsers.

Line 2 initializes the queue Q with the subparser for the initial tokenor conditional a₀, and lines 3-14 step individual subparsers until thequeue is empty, i.e., all subparsers have accepted or rejected. On eachiteration, line 4 pulls the earliest subparser p from the queue. Line 5computes the token follow-set for p.c and p.a, which contains pairs (c,a) of ordinary language tokens a and their presence conditions c. Thefollow-set computation is detailed below. Intuitively, it captures theactual variability of source code and includes the first language tokenon each path through static conditionals from the current inputposition. If the follow-set contains a single element, e.g., p.a is anordinary token and T={(p.c,p.a)}, lines 6-8 perform an LR action on theonly element T(1) and the subparser p. Unless the LR action is accept orreject, line 8 also reschedules the subparser. Otherwise, the follow-setcontains more than one element, e.g., p.a is a conditional. Since eachsubparser can only perform one LR action after the other, lines 9-12fork a subparser for each presence condition and token (c,a) ε T andthen reschedule the subparsers. Finally, line 13 tries to mergesubparsers again.

TABLE 10  1: procedure PARSE(a₀)  2:  Q.init((true, a₀, s₀)) 

 The initial subparser for a₀.  3:  while Q ≠  do  4:   p ← Q.pull( ) 

 Step the next subparser.  5;   T ← FOLLOW(p.c, p.a)  6:   if |T| = 1then  7:    

 Do an LR action and reschedule the subparser.  8:    Q.insert(LR(T(1),p))  9:   else 

 The follow-set contains several tokens 10:    

 Fork subparsers and reschedule them. 11:    Q.insertAll(FORK(T, p)) 12:  end if 13:   Q ← MERGE(Q) 14:  end while 15: end procedure

The Token Follow-Set. A critical challenge for configuration-preservingparsing is which subparsers to create. The naive strategy forks asubparser for every branch of every static conditional. But conditionalsmay have empty branches and even omit branches, like the implicit elsebranch shown in Tables 1 and 2. Furthermore, they may be directly nestedwithin conditional branches, and they may directly follow otherconditionals. Consequently, the naive strategy forks a great manyunnecessary subparsers and is intractable for complex C programs, suchas Linux. Instead, FMLR relies on the token follow-set to capture thesource code's actual variability and, thus, limits the number of forkedsubparsers.

Table 11 illustrates one example of a FOLLOW algorithm. It takes apresence condition c and a token or conditional a, and it returns thefollow-set T for a, which contains pairs (c_(i),a_(i)) of ordinarytokens a_(i) and their presence conditions c_(i). By construction, eachtoken a_(i) appears exactly once in T; consequently, the follow-set isordered by the tokens' positions in the input. Line 2 initializes T tothe empty set. Lines 3-24 define the nested procedure FIRST. It scanswell-nested conditionals and adds the first ordinary token and presencecondition for each configuration to T. It then returns the presencecondition of any remaining configuration, i.e., conditional branchesthat are empty or implicit and thus do not contain ordinary tokens.Lines 25-29 repeatedly call FIRST until all configurations have beencovered, i.e., the remaining configuration is false. Line 28 moves on tothe next token or conditional, while also stepping out of conditionals.In other words, if the token or conditional a is the last element in thebranch of a conditional, which, in turn, may be the last element in thebranch of another conditional (and so on), line 28 updates a with thefirst element after the conditionals.

TABLE 11  1: procedure FOLLOW(c, a)  2:  T ←  

 Initialize the follow-set.  3:  procedure FIRST(c, a)  4:   loop  5:   if a is a language token then  6:     T ← T ∪ {(c, a)}  7:     returnfalse  8:    else 

 a is a conditional.  9:     c_(r) ← false 

 Initialize remaining condition 10:     for all (c_(i), τ_(i)) ε a do11:      if τ_(i) = • then 12:       c_(r) ← c_(r)

 c

 c_(i) 13:      else 14:       c_(r) ← c_(r)

 FIRST(c

 c_(i), τ_(i)(1)) 15:      end if 16:     end for 17:     if c_(r) =false or a is last element in branch then 18:      return c_(r) 19:    end if 20:     c ← c_(r) 21:     a ← next token or conditional aftera 22:    end if 23:   end loop 24:  end procedure 25:  loop 26:   c ←FIRST(c, a) 27:   if c = false then return T end if 

 Done. 28:   a ← next token or conditional after a 29:  end loop 30: endprocedure

FIRST does the brunt of the work. It takes a token or conditional a andpresence condition c. Lines 4-23 then iterate over the elements of aconditional branch or at a compilation unit's top-level, starting witha. Lines 5-7 handle ordinary language tokens. Line 6 adds the token andpresence condition to the follow-set T Line 7 terminates the loop byreturning false, indicating no remaining configuration. Lines 8-22handle conditionals. Line 9 initializes the remaining configurationc_(r) to false. Lines 10-16 then iterate over the branches of theconditional a, including any implicit branch. If a branch is empty, line12 adds the conjunction of its presence condition c_(i) and the overallpresence condition c to the remaining configuration c_(r). Otherwise,line 14 recurses over the branch, starting with the first token orconditional τ_(i)(1), and adds the result to the remaining configurationc_(r). If, after iterating over the branches of the conditional, theremaining configuration is false or there are no more tokens orconditionals to process, lines 17-19 terminate FIRST's main loop byreturning c_(r). Finally, lines 20-21 set up the next iteration of theloop by updating c with the remaining configuration and a with the nexttoken or conditional.

Forking and Merging. Table 12a shows example definitions of FORK andMERGE. FORK creates new subparsers from a token follow-set T to replacea subparser p. Each new subparser has a different presence condition cand token a from the follow-set T but the same LR parser stack p.s.Consequently, it recognizes a more specific configuration than theoriginal subparser p. MERGE has the opposite effect. It takes thepriority queue Q and combines any subparsers p ε Q that are on the sametoken and have the same LR parser stack. Such subparsers are redundant:they will necessarily perform the same parsing actions for the rest ofthe input, since FMLR, like LR, is deterministic. Each merged subparserreplaces the original subparsers; its presence condition is thedisjunction of the original subparsers' presence conditions.Consequently, it recognizes a more general configuration than any of theoriginal subparsers. MERGE is similar to GLR's local ambiguity packing,which also combines equivalent subparsers, except that FMLR subparsershave presence conditions.

TABLE 12 FORK(T, p) := {(c, a, p.s) | (c, α) ε T} MERGE(Q) := {(

 p.c, a, s) | a = p.a and s = p.s ∀ p ε Q}       (a) Basic forking andmerging. FORK(T, p) := {(H, p.s) | H ε LAZY(T, p) ∪ SHARED(T, p)}LAZY(T, p) := {∪{(c, a)} | action[a, p.s] = ‘shift’ ∀(c, a) ε T}SHARED(T, p) :=      {∪{(c, a)} | action[a, p.s] = ‘reduce n’ ∀(c, a) εT}       (b) Optimized forking.

Optimizations. In addition to the token follow-set, FMLR relies on threemore optimizations to contain the state explosion caused by staticconditionals: early reduces, lazy shifts, and shared reduces. Earlyreduces provide a tie-breaker for the priority queue. When subparsershave the same head a, they favor subparsers that will reduce oversubparsers that will shift. Since reduces, unlike shifts, do not changea subparser's head, early reduces prevent subparsers from outrunningeach other and create more opportunities for merging subparsers.

While early reduces seek to increase merge opportunities, lazy shiftsand shared reduces seek to decrease the number and work of forkedsubparsers, respectively. First, lazy shifts delay the forking ofsubparsers that will shift. They are based on the observation that asequence of static conditionals with empty or implicit branches, such asthe array initializer in Table 8, often results in a follow-set, whosetokens all require a shift as the next LR action. However, since FMLRsteps subparsers by position of the head, the subparser for the firstsuch token performs its shift (plus other LR actions) and can mergeagain before the subparser for the second such token can even performits shift. Consequently, it is wasteful to eagerly fork the subparsers.Second, shared reduces reduce a single stack for several heads at thesame time. They are based on the observation that conditionals oftenresult in a follow-set, whose tokens all require a reduce to the samenon-terminal; e.g., both tokens in the follow-set of the conditional inTable 2 reduce the declaration on line 3. Consequently, it is wastefulto first fork the subparsers and then reduce their stacks in the sameway.

Table 12b formally defines examples of both lazy shifts and sharedreduces. Both optimizations result in multi-headed subparsers p :=(H,s), which have more than one head and presence condition

H:={(c ₁ ,a ₁), . . . , (c _(n) ,a _(n))}

Just as for the follow-set, each token a₁ appears exactly once in H, andthe set is ordered by the tokens' positions in the input. The FMLRalgorithm illustrated in Table 10 generalizes to multi-headed subparsersas follows. It prioritizes a multi-headed subparser by its earliest heada₁. Next, by definition of optimized forking, the follow-set of amulti-headed subparser (H,s) is H. However, the optimized version of theFMLR algorithm always performs an LR operation on a multi-headedsubparser, i.e., treats it as if the follow-set contains a singleordinary token. If the multi-headed subparser will shift, it forks off asingle-headed subparser p′ for the earliest head, shifts p′, and thenreschedules both subparsers. If the multi-headed subparser will reduce,it reduces p and immediately recalculates FORK(H,p), since the next LRaction may not be the same reduce for all heads anymore. Finally, itmerges multi-headed subparsers p if they have the same heads {(_,a₁), .. . (_,a_(n))}=p.H and the same LR parser stacks s=p.s; it computes themerged parser's presence conditions as the disjunction of the originalsubparser's corresponding presence conditions c₁=V p.H(i).c

To further illustrate FMLR, an example of using FMLR on the arrayinitializer shown in Table 8 is provided. For simplicity, NULL istreated as a token and its usual expansion to ((void *)0) is ignored.For concision, subparser and set symbols are subscripted with theircurrent line numbers in Table 8. b_(n) is used to denote the booleanvariable representing the conditional expression on line n, e.g.,

b₂˜defined(CONFIG_ACORN_PARTITION_ICS)

Finally, one iteration through FMLR's main loop in Table 8 is referredto as a step.

Since Table 8 line 1 contains only ordinary tokens, FMLR behaves like anLR parser, stepping through the tokens with a single subparser p₁. Uponreaching line 2, FMLR computes FOLLOW for the conditional on lines 2-4.To this end, FIRST iterates over the conditionals and NULL token in theinitializer list by updating a in Table 11, line 21. On each iterationbesides the last, FIRST also recurses over the branches of aconditional, including the implicit else branch. As a result, it updatesthe remaining configuration in Table 11, line 12 with a conjunction ofnegated conditional expressions, yielding the follow-set:

T ₂={9 b ₂, adfspart_check_ICS), (

b₂

b₅, adfspart_check_POWERTEC), (

b₂

b₅

b₈

. . . , NULL)}

Since all tokens in T₂ reduce the empty input to the InitializerListnonterminal, shared reduces turns p₂ into a multi-headed subparser withH₂=T₂ FMLR then steps p₃. It reduces the subparser, which does notchange the heads, i.e., H₃=T₃, but modifies the stack to:

p ₂ ·s= . . . {InitializerList

It then calculates FORK(H₃,T₃); since all tokens in H₃ now shift, lazyshifts produces the same multi-headed subparser. FMLR steps p₃ again. Itforks off a single-headed subparser p′₃ and shifts the identifier tokenon line 3 onto its stack. Next, FMLR steps p′₃. It shifts the commatoken onto the stack, which yields:

p′ ₃ ·s= . . . {InitializerList adfspart_check_ICS,

and updates the head p′₃·a to the conditional on lines 5-7. FMLR stepsp′₃ again, computing the subparser's follow-set as:

T′ ₅={(b ₂

b ₅, adfspart_check_POWERTEC), . . . , (b ₂

b ₅

b ₈

. . . , NULL)}

Since all tokens in T′₅ reduce the top three stack elements to anInitializerList, shared reduces turns p′₅ into a multi-headed subparserwith H₅=T₅. At this point, both p₆ and p′₆ are multi-headed subparserswith the same heads, though their stacks differ. Due to early reduces,FMLR steps p′₆. It reduces the stack, which yields the same stack asthat of p₆, and calculates FORK, which does not change p′₆ due to lazyshifts. It then merges the two multi-headed subparsers, which disjoinsb₂ with

b₂ for all presence conditions and thus eliminates b₂ from H₆. FMLR thenrepeats the process of forking, shifting, reducing, and merging for theremaining 17 conditionals until a single-headed subparser p completesthe array initializer on lines 12-13. That way, FMLR parses 2¹⁸ distinctconfigurations with only 2 subparsers.

Building Abstract Syntax Trees. To simplify AST construction,implementations can includes an annotation facility that eliminatesexplicit semantic actions in most cases. Developers can simply addspecial comments next to productions. An AST tool can then extract thesecomments and generate the corresponding parser plug-in code, which isinvoked when reducing a subparser's stack. By default, implementationsof the invention can create an AST node that is an instance of a genericnode class, is named after the production, and has the semantic valuesof all terminals and non-terminals as children. Four annotationsoverride this default. First, layout omits the production's value fromthe AST. It is used for punctuation. Second, passthrough reuses achild's semantic value, if it is the only child in an alternative. It isparticularly useful for expressions, whose productions tend to be deeplynested for precedence (17 levels for C). Third, list encodes thesemantic values of a recursive production as a linear list. It isnecessary because LR grammars typically represent repetitions asleft-recursive productions. Fourth, action executes arbitrary programcode instead of automatically generating an AST node.

A fifth annotation, complete, determines which productions are completesyntactic units. Implementations can merge only subparsers with thesame, complete non-terminal on top of their stacks; while merging, theycan combine the subparsers' semantic values with a static choice node.The selection of complete syntactic units requires care. Treating toomany productions as complete forces downstream tools to handle staticchoice nodes in too many different language constructs. Treating too fewproductions as complete may result in an exponential subparser number inthe presence of embedded configurations, e.g., the array initializer inTable 8. Implementations of the invention's C grammar try to strike abalance by treating not only declarations, definitions, statements, andexpressions as complete syntactic units, but also members in commonlyconfigured lists, including struct and union members, functionparameters, and struct, union, and array initializers.

Managing Parser Context. In addition, implementations can include acontext management plug-in that enables the recognition ofcontext-sensitive languages, including C, without modifying the FMLRparser. The plug-in can have four callbacks: (1) reclassify modifies thetoken follow-set by changing or adding tokens. It is called aftercomputing the follow-set, i.e., line 5 in Table 10. (2) forkContextcreates a new context and is called during forking. (3) mayMergedetermines whether two contexts allow merging and is called whilemerging subparsers. (4) mergeContexts actually combines two contexts andis also called while merging.

An example plug-in for parsing C can work as follows. Its context is asymbol table, which tracks which names denote values or types underwhich presence conditions and in which C language scopes. Productionsthat declare names and enter/exit C scopes update the symbol tablethrough helper productions that are empty but have semantic actions.reclassify checks the name of each identifier. In one implementation,these are the only token generated for names. If the name denotes a typein the current scope, reclassify replaces the identifier with a typedefname. If the name is ambiguously defined under the current presencecondition, it instead adds the typedef name to the follow-set. Thiscauses the FMLR parser to fork an extra subparser on such names, eventhough there is no explicit conditional. forkContext duplicates thecurrent symbol table scope. mayMerge allows merging only at the samescope nesting level. Finally, mergeContexts combines any symbol tablescopes not already shared between the two contexts.

Error Handling. Some branches of static conditionals may containmalformed preprocessor operations or malformed language code. Unlike aregular preprocessor, the configuration-preserving preprocessor does nothalt after encountering a malformed preprocessor operation. Instead, itpasses a special token to the parser that indicates an erroneousconditional branch. The configuration-preserving parser, in turn, doesnot parse such erroneous conditional branches but instead adds a specialerror node to the AST. The referring static choice node identifies theconditional branch's presence condition, while the error node identifiesthe cause and location of the error. Similarly, if theconfiguration-preserving parser encounters malformed language code, itstops parsing of the conditional branch and instead creates an errornode in the AST. Again, the referring static choice node identifies thebranch's presence condition, while the error node identifies the causeand location of the error.

EXAMPLES

The described configuration-preserving preprocessor and parser can beunderstood more readily by reference to the following example, which areprovided by way of illustration and are not intended to be limiting inany way.

Preprocessor Usage and Interactions. Table 13 provides a developer'sview of preprocessor usage in the x86 Linux kernel. The data wascollected by running cloc, grep, and we on individual C and headerfiles. Table 13a compares the number of preprocessor directives to linesof code (LoC), excluding comments and empty lines. Even this simpleanalysis demonstrates extensive preprocessor usage: almost 10% of allLoC are preprocessor directives. Yet, when looking at C files,preprocessor usage is not nearly as evident for two reasons. First,macro invocations look like C identifiers and C function calls; they mayalso be nested in other macros. Consequently, they are not captured bythis analysis. Second, C programs usually rely on headers for commondefinitions, i.e., as a poor man's module system. The data corroboratesthis. 66% of all directives and 84% of macro definitions are in headerfiles. Furthermore, 16% of include directives are in header files,resulting in long chains of dependencies. Finally, some headers aredirectly included in thousands of C files (and preprocessed for eachone). Table 2b shows the top five most frequently included headers;module.h alone is included in nearly half of all C files.

TABLE 13 (a) Number of directives compared to lines of code (LoC). TotalC Files Headers LoC 5,632,708 85% 15% All Directives 531,528 34% 66%#define 364,326 16% 84% #if, #ifdef, #ifndef 38,297 58% 42% #include87,280 84% 16% (b) The top five most frequently included headers. HeaderName C Files That Include Header include/linux/module.h 3,741 (49%)include/linux/init.h 2,841 (37%) include/linux/kernel.h 2,567 (33%)include/linux/slab.h 1,800 (23%) include/linux/delay.h 1,505 (20%)

Table 14 provides a tool's view of preprocessor usage in the x86 Linuxkernel. The data was collected by instrumenting an implementation of theinvention and applying it on compilation units, i.e., C files plus theclosure of included headers. It captures information not available inthe simple counts of Table 13, including macro invocations. Table 14loosely follows the organization of Table 3. Each row shows apreprocessor or C language construct. The first column names theconstruct, the second column shows its usage, and the third and fourthcolumns show its interactions. Each entry is the distribution in threepercentiles, “50th·90th·100th,” across compilation units. Table 14confirms that preprocessor usage is extensive. It also confirms thatmost interactions identified above occur in real-world C code.

TABLE 14 Language Construct Total Interaction with Conditionals OtherInteractions Macro Definitions 34k · 45k · 122k Contained in 34k · 45k ·122k Redefinitions 23k · 33k · 111k Macro Invocations 98k · 140k · 381kTrimmed 16k · 21k · 70k Nested invocations 64k · 97k · 258k Hoisted 154· 292 · 876 Built-in macros 135 Token-Pasting 4k · 6k · 22k Hoisted 0 ·0 · 180 Stringification 6k · 8k · 23k Hoisted 361 · 589 · 6,082 FileIncludes 1,608 · 2,160 · 5,939 Hoisted 33 · 55 · 165 Computed includes34 · 56 · 168 Reincluded headers 1,184 · 1,744 · 5,488 StaticConditionals 8k · 10k · 29k Hoisted 331 · 437 · 1,258 With non-boolean509 · 713 · 1,975 Max. depth 28 · 33 · 40 expressions Error Directives42 · 57 · 168 C Declarations & 34k · 49k · 127k Containing X · X · XStatements Typedef Names 734 · 1,010 · 2,554 Ambiguously 0 · 0 · 0defined names

The vast majority of measured preprocessor interactions involve macros.First, almost all macro definitions are contained in staticconditionals, i.e., any difference is hidden by rounding to the nearestthousand. This is due to most definitions occurring in header files andmost header files, in turn, containing a single static conditional thatprotects against multiple inclusion. Second, over 60% of macroinvocations appear from within other macros; e.g., the median for totalmacro invocations is 98k, while the median for nested invocations is64k. This makes it especially difficult to fully analyze macroinvocations without running the preprocessor, e.g., by inspecting sourcecode. While not nearly as frequent as interactions involving macros,static conditionals do appear within function-like macro invocations,token-pasting and stringification operators, file includes, as well asconditional expressions. Consequently, a configuration-preservingpreprocessor must hoist such conditionals. Similarly, non-booleanexpressions do appear in conditionals and the preprocessor must preservethem. However, two exceptions are notable. Computed includes are veryrare and ambiguously-defined names do not occur at all, likely becauseboth make it very hard to reason about source code.

Subparser Counts. According to Table 14, most compilation units containthousands of static conditionals. This raises the question of whetherrecognizing C code across conditionals is even feasible. Two factorsdetermine feasibility: (1) the breadth of conditionals, which forces theforking of subparsers, and (2) the incidence of partial C constructs inconditionals, which prevents the merging of subparsers. The number ofsubparsers per iteration of FMLR's main loop in Table 10, lines 3-14precisely captures the combined effect of these two factors.

Table 15 and FIG. 2 shows the cumulative distribution of subparsercounts per FMLR iteration for the x86 Linux kernel under differentoptimization levels: Table 15 identifies the maxima and FIG. 2characterizes the overall shape. Table 15 also includes MAPR, a toolthat includes a C preprocessor and parser, as a baseline. Table 15demonstrates that MAPR is intractable for Linux, triggering akill-switch at 16,000 subparsers for 98% of all compilation units. Incontrast, the token follow-set alone makes FMLR feasible for the entirex86 Linux kernel. The lazy shifts, shared reduces, and early reducesoptimizations further decrease subparser counts, by up to a factor 5.8.They also help keep the AST smaller: fewer forked subparsers means fewerstatic choice nodes in the tree, and earlier merging means more treefragments outside static choice nodes, i.e., shared betweenconfigurations.

TABLE 15 Subparsers Optimization Level 99th % Max. Shared, Lazy, & Early20 39 Shared & Lazy 21 39 Shared 20 77 Lazy X X Follow-Set FMLR 32 227 MAPR & Largest First >16,000 on 98% of comp. units MAPR >16,000 on 98%of comp. units

Performance. In this example, the tested implementation ran on the Javavirtual machine. TypeChef, a variability aware preprocessor and parser,was also run on the Java virtual machine, which enables a directperformance comparison. All of the tested implementation and TypeChef'spreprocessor are written in Java, whereas TypeChef's parser is writtenin Scala. Running either tool on x86 Linux requires some preparation.(1) both tools were configured with gcc's built-in macros. The testedimplementation automated this through its build system; TypeChef'sdistribution includes manually generated files for different compilersand versions. (2) Both tools required a list of C files identifying thekernel's compilation units. The list of 7,665 C files distributed withTypeChef was used. (3) The tested implementation was configured withfour definitions of non-boolean macros, which were discovered bycomparing the result of running gcc's preprocessor, i.e., gcc -E, underthe allyesconfig configuration on the 7,665 C files with the result ofrunning it on the output of the tested implementation'sconfiguration-preserving preprocessor for the same files. With thosefour definitions in place, the results were identical modulo whitespace.(4) TypeChef needs to be configured with over 300 additional macrodefinitions. It also treats macros that are not explicitly marked asconfiguration variables, i.e., have the CONFIG_prefix, as undefinedinstead of free.

Two different configurations were run based upon the preparation stepsmentioned above. In one test, only the first three preparation stepswere done. This was considered the unconstrained kernel. In anothertest, all four preparation steps were done, and this was called theconstrained kernel. At the time of this example, TypeChef ran only onthe constrained kernel, and only on version 2.6.33.3. To ensure thatresults are comparable, the examples and experiments also used version2.6.33.3 of Linux. At the same time, the tested implementation ran onboth constrained and unconstrained kernels. In fact, the data presentedin Table 3 for preprocessor usage and in Table 15 for subparser countswas collected by running the tested implementation on the unconstrainedkernel. The constrained kernel has less variability: its 99^(th) and100^(th) percentile subparser counts are 12 and 32, as opposed to 21 and39 for the unconstrained kernel. The tested implantation was also ran onother versions of Linux; we validated our tool on the latest stableversion, 3.2.9.

FIG. 3 shows the cumulative latency distribution across compilationunits of the constrained kernel of the tested implementation andTypeChef when ran on an off-the-shelf PC. For each tool, it alsoidentifies the maximum latency for a compilation unit and the totallatency for the kernel. The latter number should be treated as aconvenient summary, but no more: workload and tools easily parallelizeacross cores and machines. When considering the 50th and 80^(th)percentiles, both tools perform reasonably well. While the testedimplementation was between 3.4 to 3.8 times faster than TypeChef, bothcurves show a mostly linear increase, which is consistent with a normaldistribution. However, the “knee” in TypeChef's curve at about 25seconds and the subsequent long tail, reaching over 15 minutes,indicates a serious scalability bottleneck.

FIG. 4 plots the breakdown of the latency of the tested implementation.It demonstrates that the performance of the tested implementation scaledroughly linearly with compilation unit size and lexing, preprocessing,and parsing each scale roughly linearly as well. Furthermore, most ofthe total latency was split between preprocessing and parsing. Toprovide a performance baseline, the cumulative latency distribution forgcc lexing, preprocessing, and parsing the 7,665 compilation units underallyeseonfig was measured. The timing data was gathered usinggcc's-ftime-report command line option. The 50th, 90th, and 100^(th)percentiles are 0.18, 0.24, and 0.87 seconds, i.e., a factor 12-32speedup compared to the tested implementation. It reflects that gcc doesnot have to preserve static conditionals and that gcc's C implementationhas been carefully tuned for many years.

FIG. 5 is a block diagram of a computer system in accordance with anillustrative implementation. The computer system or computing device 500can be used to implement a configuration-preserving preprocessor andparser in accordance with one or more implementations of the presentinvention. The computing system 500 includes a bus 505 or othercommunication component for communicating information and a processor510 or processing circuit coupled to the bus 505 for processinginformation. The computing system 500 can also include one or moreprocessors 510 or processing circuits coupled to the bus for processinginformation. The computing system 500 also includes main memory 515,such as a random access memory (RAM) or other dynamic storage device,coupled to the bus 505 for storing information, and instructions to beexecuted by the processor 510. Main memory 515 can also be used forstoring position information, temporary variables, or other intermediateinformation during execution of instructions by the processor 510. Thecomputing system 500 may further include a read only memory (ROM) 510 orother static storage device coupled to the bus 505 for storing staticinformation and instructions for the processor 510. A storage device525, such as a solid state device, magnetic disk or optical disk, iscoupled to the bus 505 for persistently storing information andinstructions.

The computing system 500 may be coupled via the bus 505 to a display535, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 530, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 505 for communicating information and command selections to theprocessor 510. In another implementation, the input device 530 has atouch screen display 535. The input device 530 can include a cursorcontrol, such as a mouse, a trackball, or cursor direction keys, forcommunicating direction information and command selections to theprocessor 510 and for controlling cursor movement on the display 535.

According to various implementations, the processes described herein canbe implemented by the computing system 500 in response to the processor510 executing an arrangement of instructions contained in main memory515. Such instructions can be read into main memory 515 from anothercomputer-readable medium, such as the storage device 525. Execution ofthe arrangement of instructions contained in main memory 515 causes thecomputing system 500 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory515. In alternative implementations, hard-wired circuitry may be used inplace of or in combination with software instructions to effectillustrative implementations. Thus, implementations are not limited toany specific combination of hardware circuitry and software.

Although an example computing system has been described in FIG. 5,implementations described in this specification can be implemented inother types of digital electronic circuitry, or in computer software,firmware, or hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them.

Implementations described in this specification can be implemented indigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.The implementations described in this specification can be implementedas one or more computer programs, i.e., one or more modules of computerprogram instructions, encoded on one or more computer storage media forexecution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded onan artificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate components or media (e.g., multiple CDs, disks, or otherstorage devices). Accordingly, the computer storage medium is bothtangible and non-transitory.

The operations described in this specification can be performed by adata processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “data processing apparatus” or “computing device” encompassesall kinds of apparatus, devices, and machines for processing data,including by way of example a programmable processor, a computer, asystem on a chip, or multiple ones, or combinations of the foregoing.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features described in this specification in thecontext of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresdescribed in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and tables in aparticular order, this should not be understood as requiring that suchoperations be performed in the particular order shown or in sequentialorder, or that all illustrated operations be performed, to achievedesirable results. In certain circumstances, multitasking and parallelprocessing may be advantageous. Moreover, the separation of varioussystem components in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated in a single software product or packagedinto multiple software products.

Thus, particular implementations of the invention have been described.Other implementations are within the scope of the following claims. Insome cases, the actions recited in the claims can be performed in adifferent order and still achieve desirable results. In addition, theprocesses depicted in the accompanying figures do not necessarilyrequire the particular order shown, or sequential order, to achievedesirable results. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A system comprising: a lexer configured togenerate a plurality of tokens from one or more source files, the one ormore source files including source code in a first programming language,the source code comprising one or more static conditionals that includea conditional expression and branch code that is operative when theconditional expression is true, and wherein various configurations arepossible based upon the conditionals; and a configuration-preservingpreprocessor configured to: determine that a first static conditionalincludes one or more nested static conditionals within the branch codeassociated with the first static conditional; and hoist each of the oneor more nested static conditionals to a beginning of the branch codeassociated with the first static conditional, wherein each innermostbranch code does not contain a static conditional, and wherein eachpossible configuration is preserved.
 2. The system of claim 1, whereinthe configuration-preserving preprocessor is further configured toconvert each conditional expression into a binary function.
 3. Thesystem of claim 2, wherein the configuration-preserving preprocessor isfurther configured to convert each static conditional into a member of astatic conditional set, and wherein the member comprises the respectivebinary function and tokens associated with the respective branch code.4. The system of claim 2, wherein the configuration-preservingpreprocessor is further configured to store a macro definition with anassociated binary function in a macro table, wherein the binary functionindicates when the macro is valid.
 5. The system of claim 4, wherein theconfiguration-preserving preprocessor is further configured to removeinfeasible entries from the macro table.
 6. The system of claim 2,wherein the binary function is a binary decision diagram.
 7. A methodcomprising: generating a plurality of tokens from one or more sourcefiles, the one or more source files including source code in a firstprogramming language, the source code comprising one or more staticconditionals that include a conditional expression and branch code thatis operative when the conditional expression is true, and whereinvarious configurations are possible based upon the conditionals;determining that a first static conditional includes one or more nestedstatic conditionals within the branch code associated with the firststatic conditional; and hoisting, using a processor, each of the one ormore nested static conditionals to a beginning of the branch codeassociated with the first static conditional, wherein each innermostbranch code does not contain a static conditional, and wherein eachpossible configuration is preserved.
 8. The method of claim 7, furthercomprising converting each conditional expression into a binaryfunction, wherein the binary function is a function that takes twoinputs.
 9. The method of claim 8, further comprising converting eachstatic conditional into a member of a static conditional set, andwherein the member comprises the respective binary function and tokensassociated with the respective branch code.
 10. The method of claim 8,further comprising storing a macro definition with an associated binaryfunction in a macro table, wherein the binary function indicates whenthe macro is valid.
 11. The method of claim 10, further comprisingremoving infeasible entries form the macro table.
 12. The method ofclaim 8, wherein the binary function is a binary decision diagram.
 13. Anon-transitory computer-readable medium having instructions storedthereon, the instructions comprising: instructions to generate aplurality of tokens from one or more source files, the one or moresource files including source code in a first programming language, thesource code comprising one or more static conditionals that include aconditional expression and branch code that is operative when theconditional expression is true, and wherein various configurations arepossible based upon the conditionals; instructions to determine that afirst static conditional includes one or more nested static conditionalswithin the branch code associated with the first static conditional; andinstructions to hoist each of the one or more nested static conditionalsto a beginning of the branch code associated with the first staticconditional, wherein each innermost branch code does not contain astatic conditional, and wherein each possible configuration ispreserved.
 14. The non-transitory computer-readable medium of claim 13,further comprising converting each conditional expression into a binaryfunction, wherein the binary function is a function that takes twoinputs.
 15. The non-transitory computer-readable medium of claim 14,further comprising converting each static conditional into a member of astatic conditional set, and wherein the member comprises the respectivebinary function and tokens associated with the respective branch code.16. The non-transitory computer-readable medium of claim 14, furthercomprising storing a macro definition with an associated binary functionin a macro table, wherein the binary function indicates when the macrois valid.
 17. The method of claim 16, further comprising removinginfeasible entries form the macro table.
 18. The non-transitorycomputer-readable medium of claim 14, wherein the binary function is abinary decision diagram.